Repository navigation
Conversation
Default compression in Cutout.prepare is now zstd (complevel 3) after BitRound quantization to 14 mantissa bits (max relative error 2**-15). The encoding applies to all variables, because xarray < 2026.9 drops the zstd setting of existing variables when the cutout file is rewritten. Require netCDF4>=1.7.4, whose wheels bundle the zstd plugin. Closes #508
euronion
left a comment
There was a problem hiding this comment.
Interesting.
Did you compare the old and new cutout values as well, to be sure there is not further corruption/mix of values from e.g. NaN to 0?
We do not have the atlite version encoded somewhere in the cutouts, do we? So we cannot mark new cutouts incompatible with older versions of the library, can we? A feature we should implement.
Yes, they were within numerical tolerances. Although I will have to look at the failing CI tests for older Python versions.
No, we don't have that yet. And yes, we should. |
Closes #508.
Changes proposed in this Pull Request
Cutout.preparenow compresses cutouts with zstd (complevel 3) after BitRound quantization to 14 mantissa bits (Klöwer et al., 2021). The maximum relative error is 2**-15 = 3.1e-5.{"compression": "zstd", "complevel": 3, "quantize_mode": "BitRound", "significant_digits": 14}. For the old behaviour, pass{"compression": "zlib", "complevel": 9, "shuffle": True}.preparecall would store the earlier variables uncompressed.netCDF4>=1.7.4, whose wheels bundle the zstd plugin. conda-forgelibnetcdf(tested with 4.10.1) also works.Benchmark
Europe, January to March 2013, 2160×131×177, 3.2 GB in memory:
Compatibility
preparecall recompresses them with zstd.netCDF4<1.7.4(pip) cannot read the new cutouts. It fails withNetCDF: Filter error: undefined filter encountered.Checklist
doc.environment.yaml,environment_docs.yamlandsetup.py(if applicable).doc/release_notes.rstof the upcoming release is included.